Picked-up image display method

ABSTRACT

An image display method in which a position of a shot image of the ground surface having been taken from the air is specified three-dimensionally, a photographic area on the ground surface having been shot is obtained by computation, and a shot image is transformed in conformity with the photographic area thereof and thereafter displayed being superposed on a map of geographic information system, and in which landmarks are extracted from a map of a geographic information system and a shot image respectively, and the corresponding landmarks are compared, whereby a parameter for use in computing a photographic area having been shot is compensated.

TECHNICAL FIELD

The present invention relates to an image display method characterizedin that an image, which is transmitted from a photographic devicemounted onto, for example, a helicopter, is displayed being superposedon a map of a geographic information system, thereby enabling todetermine situations on the ground easily as well as with sufficientprecision in the case where natural disaster such as earthquake or fire,or human disaster such as explosion or serious accident occur.

BACKGROUND ART

In the conventional positional specification method and device, asshown, for example, in the Japanese Patent No.2695393, a shootingposition in the air is specified three-dimensionally, a direction of atarget with respect to a shooting position is measured, a ground surfacewhere the target resides is obtained based on a three-dimensionaltopographic data including altitude information as to undulation of theground surface which data has been preliminarily prepared, and aposition of the target on the ground surface having been shot from theair is specified as a point of intersection of the ground surface with astraight line extending from the shooting position toward the target.

In the conventional positional specification method and device, tospecify the position of a target on the ground surface, athree-dimensional topographic data including altitude information as toundulation of the ground surface which data has been preliminarilyprepared is needed as a prerequisite. Further, measurement error thatoccurs at the time of specifying three-dimensionally a shooting positionin the air and at the time of measuring the direction of a target withrespect to the shooing position cannot be compensated, thus making ithard to specify a position with accuracy. Furthermore, since thepositional specification is executed with respect to one point oftarget, a problem exists in that situations on the ground surface cannotbe understood area-wide.

DISCLOSURE OF INVENTION

The present invention was made to solve the above-discussed problems,and has an object of providing an image display method in which shotimages are displayed being superposed on a map of a geographicinformation system, thereby enabling to understand area-wide situationson the ground surface having been shot; as well as in which a displayposition on the map of an image is compensated by comparison between theshot image and the map to carry out the superposed display with highprecision, thereby enabling to understand situations of the groundsurface having been shot more easily and rapidly. To accomplish theforegoing objects, an image display method according to the invention isa method of image processing and displaying a shot image of the groundsurface having been taken with photographic equipment that is mounted onan airframe in the air, in which a shooting position in the air isspecified three-dimensionally, a photographic area on the ground surfacehaving been shot is obtained by computation, and a shot image istransformed in conformity with the mentioned photographic area andthereafter displayed being superposed on a map of a geographicinformation system.

A further image display method is a method of image processing anddisplaying a shot image of the ground surface having been taken withphotographic equipment that is mounted on an airframe in the air, inwhich a shooting position in the air is specified three-dimensionally, aphotographic area of the ground surface having been shot is obtained bycomputation, and a shot image is transformed in conformity with thementioned photographic area and thereafter displayed being superposed ona map of a geographic information system; and in which landmarks areextracted from a map of the geographic information system and a shotimage respectively and the corresponding landmarks are compared, wherebya parameter for use in computing a photographic area of the groundsurface having been shot is compensated, and a shot image is displayedbeing superposed with high precision on a map of the geographicinformation system.

According to this invention, it becomes easy to ascertain conformitybetween image information and a map, thereby enabling to identify atarget point of land easily.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a system for carrying out an imagedisplay method according to a first preferred embodiment of the presentinvention.

FIG. 2 is an explanatory diagram of functions of map processing means inthe first embodiment.

FIG. 3 is a photograph showing a display screen according to the firstembodiment.

FIG. 4 is a photograph showing a display screen obtained by an imagedisplay method according to a second embodiment of the invention.

FIG. 5 are views explaining a third embodiment of the invention.

FIG. 6 are diagrams explaining map processing in the third embodiment.

FIG. 7 are views explaining a fourth embodiment of the invention.

FIG. 8 are diagrams explaining map processing in the fourth embodiment.

FIG. 9 are views explaining a fifth embodiment of the invention.

FIG. 10 are diagrams explaining map processing in the fifth embodiment.

FIG. 11 is a diagram for explaining map processing of an image displaymethod according to a sixth embodiment of the invention.

FIG. 12 is a view for explaining map processing of an image displaymethod according to a seventh embodiment of the invention.

FIG. 13 are views explaining an image display method according to aneighth embodiment of the invention.

FIG. 14 is a block diagram showing a system for carrying out an imagedisplay method according to a ninth embodiment of the invention.

FIG. 15 is an explanatory diagram of functions of map processing meansin the ninth embodiment.

FIG. 16 is a flowchart showing operations in the image display methodaccording to the ninth embodiment.

FIG. 17 are views explaining angle parameters for use in computing aphotographic frame in map processing means according to the ninthembodiment.

FIG. 18 are diagrams explaining the photographic frame computation inmap processing means according to the ninth embodiment.

FIG. 19 is a diagram explaining parameter compensation in map processingmeans according to the ninth embodiment.

FIG. 20 are views showing effects in the image display method accordingto the ninth embodiment.

FIG. 21 are views explaining an eleventh embodiment of the invention.

FIG. 22 are diagrams explaining a twelfth embodiment of the invention.

FIG. 23 is a flowchart showing operations in an image display methodaccording to a fourteenth embodiment of the invention.

FIG. 24 is a view showing effects in the image display method accordingto the fourteenth embodiment.

FIG. 25 are diagrams explaining a fifteenth embodiment of the invention.

FIG. 26 is a view explaining a sixteenth embodiment of the invention.

BEST MODE FOR CARRYING OUT THE INVENTION Embodiment 1

First, the present invention is summarized. The invention is to displaya shot image of the ground having been shot from the air, beingsuperposed on a map of a geographic information system (GIS=GeographicInformation System, system of displaying a map on the computer screen),thereby making it easy to acknowledge conformity between an imageinformation and a map, and making it easy to determine a target point ofland. In this regard, in the case of taking a shot of the ground fromthe air with a camera, an image thereof is taken only in a certainrectangular shape at all times regardless of direction of the camera.Therefore, it is difficult to superpose (paste) as it is an image havingbeen shot on a map that is obtained with the geographic informationsystem. Thus, according to this invention, a photographic area(=photographic frame) of the ground surface to be shot, the photographicarea complicatedly varying from a rectangle to a shape close totrapezoid or rhombus depending on, e.g., posture of the camera withrespect to the ground, is obtained by computation using camerainformation and posture information of an airframe at the time ofshooting an image. Then the shot image is transformed in conformity withthe image frame, pasted onto the map, and displayed.

Hereinafter, an image processing method and an image processing systemaccording to a first preferred embodiment of the invention is describedwith reference to the drawings. FIG. 1 is a block diagram explaining asystem of carrying out the method of the invention. FIG. 2 is a diagramexplaining functions of map processing means. The method of theinvention is implemented with an on-board system 100 formed of a flightvehicle (=airframe) such as helicopter on which, e.g., photographicequipment (=camera) is mounted, and a ground system 200 located on theground that receives signals from the on-board system 100 and processesthem. The on-board system 100 is formed of on-board devices includingphotographic means for taking a shot of the ground from the air,airframe position measurement means 108 or airframe posture measurementmeans 107 acting as information collection section that obtainsinformation for specifying three-dimensionally a shooting position ofphotographic means, and transmission means for transmitting a shot imagehaving been taken by the mentioned photographic means and informationobtained by the mentioned information collection section.

More specifically, on the on-board system 100, a camera 102 acting asphotographic means 105 that takes pictures of the ground from the air ismounted. The airframe 101 is provided with airframe position measurementmeans 108 that obtains current positional information with an antenna103, being a GPS signal receiving section, and detects an airframeposition, and a gyro. The airframe 101 is further provided with airframeposture measurement means 107 that performs airframe posture detectionof detecting a posture, that is, an elevation angle (=pitch) and a rollangle of the airframe 101.

The photographic means 105 including the camera 102 takes a shot of theground and outputs image signals thereof, and also outputs camerainformation such as diaphragm or zoom of the camera as well. The camera102 is attached to a gimbal, and this gimbal includes camera posturemeasurement means 106 detecting a rotation angle (=pan) and inclination(=tilt) of the camera, and outputs values thereof.

An output signal from the mentioned airframe position measurement means108, an output signal from the mentioned airframe posture measurementmeans 107, an image signal and a camera information signal of thementioned camera shooting means 105, an output signal from the mentionedcamera posture measurement means 106 are multiplexed and modulated bymultiplex modulator 109. These signals are converted to digital signalsby signal conversion means 110, and transmitted to the ground system 200from transmission means 104 having tracking means 111.

The ground system 200 is mainly constituted of: an input section thatinputs a shot image of the ground surface, which photographic meanstakes from the air, and information for three-dimensionally specifying ashooting position of the above-mentioned photographic means; a signalprocessing section that performs signal processing with respect toinformation having been inputted; a geographic information system thatdisplays a map on the screen; and a map processing section thatprocesses the image as well as the information having been processed atthe signal processing section, and displays the resultant picture on themonitor.

More specifically, signals from the on-board system 100 are receivedwith receiving means 201 including tracking means 202, andsignal-converted by signal conversion means 203. These signals arefetched out as image signals and the other information signals such asairframe position, airframe posture, camera posture or camerainformation with multiplex demodulator 204. These fetched-out signalsare signal-processed with signal processing means 205, and the imagesignals are used in map processing with map processing means 206 in thenext step as a moving image data 207 and a still image data 208. Otherinformation signals including a two-dimensional map data 209 and atopographic data 210 of the geographic information system are also usedin map processing with map processing means 206. Numeral 211 designatesmonitor display means.

FIG. 2 is a schematic diagram showing map processing means of the imagedisplay system according to this first embodiment. The map processingmeans 206, as shown in FIG. 2, executes the processing with a movingimage data 207 and still image data 208, being image signals,information signals of airframe position, airframe posture and cameraposture, and a two-dimensional map data 209 and a three-dimensionaltopographic data 210 of the geographic information system. This mapprocessing means 206 is mainly constituted of a photographic areacomputing section (image frame computing 212) that obtains aphotographic area on the map of the geographic information systemcorresponding to a photographic area of a shot image, which thephotographic means has taken; an image transformation section (imagetransformation 213) that transforms the mentioned shot image inconformity with a photographic area having been obtained by the imageframe computing 212; and a monitor (e.g., super impose 214) thatdisplays the mentioned transformed shot image super imposed on thementioned photographic area of the mentioned map.

At the map processing 206, first, image frame computation is executed inimage frame 212 in which a shooting position in the air is specifiedthree-dimensionally with information signals regarding an airframeposition, and a photographic area (=photographic frame) of the groundsurface having been shot is obtained by computation based on posture ofthe camera and airframe with respect to the ground surface. Imagetransformation 213 is performed in conformity with this image frame.This image transformation is to transform the image so that an imagebecomes, e.g., a shape close to trapezoid, or rhombus in which shape theimage conforms to the map. Then, the transformed image is superposed(pasted) in superposition step 214 onto a map of the geographicinformation system. Thereafter, this resultant picture is displayed withmonitor display means 211 such as CRT.

FIG. 3 is a photograph in which a shot image 302 is superposed on a map301 of the geographic information system with a photographic frame 303corresponding to the map. Numeral 304 designates a flight path of theairframe, and numeral 305 designates an airframe position (cameraposition). The map processing including the above-describedtransformation processing with the map processing means 206 causes animage to be in coincidence with the map substantially at all points, asshown in FIG. 3, and makes it easy to ascertain conformity between imageinformation and map, thereby enabling to determine a target pointeasily.

Further, as shown in FIG. 3, an image of the image frame having beenshot with the camera, can be displayed being superposed on the map, aswell as it can be done easily to erase the shot image 302 and displayonly the image frame 303. Herein the shot image 302 is superposed on thetwo-dimensional map. Accordingly, for example, a place of the disasteroccurrence (e.g., building on fire) is visually confirmed with the shotimage 302, and the position thereof is checked (clicked) on the shotimage 302. Thereafter, the image 302 is erased, and the two-dimensionalmap under the shot image 302 is displayed leaving only the image frame303 displayed, thus enabling to rapidly recognize a place on the map ofthe position having been checked on the shot image. Further, supposingthat displayed images on a monitor are arranged to display in a definitedirection regardless of a direction of the camera, the determination ofa target point becomes still easier.

Embodiment 2

According to this second embodiment, a current position of the airframe101 is measured, a photographic frame of the ground having been shotfrom on board is computed, and an image having been shot is transformedand pasted onto a map of the geographic information system in conformitywith the photographic frame. At the time of executing a comparisonbetween a shot image and a map is done, plural pieces of shot images aresampled in succession in cycles of a predetermined time period fromimages having been continuously shot. Then a series of plural images arepasted onto the map of the geographic information system to bedisplayed, and a target point of land is specified from the imagespasted onto the map.

FIG. 4 shows a monitor display screen according to this method. Numeral304 designates a flight path of the airframe. Numeral 305 designates anairframe position (camera position) Images having been shot with thecamera along the flight path 304 are sampled with a predetermined timingto obtain each image frame, and the shot images are transformed andprocessed so as to conform to the image frames and pasted onto the map301. Numerals 302 a to 302 f are pasted images. Numerals 303 a to 303 fare image frames thereof.

The computation of a photographic frame and the transformation of animage into each image frame are executed by computing with the use ofcamera information and posture information of the airframe at the timeof taking a shot as described in the first embodiment. It is preferablethat a sampling period for each image is changed in accordance with aspeed of the airframe. Normally, a sampling period is set to be shorterwhen the airframe flies at high speed, and the sampling period is set tobe longer when the airframe flies at low speed.

According to this second embodiment, it becomes possible to identifysituations on the ground while confirming the situations of a wide rangeof ground surface with a map and plural pieces of continuous images,thereby enabling to determine a target point of land more effectively.

Embodiment 3

According to this third embodiment, a current position of the airframe101 and a rotation angle and inclination (pan and tilt=posture of thecamera) of the camera 102 with respect to the airframe are measured, anda photographic frame of the ground having been shot from on board iscomputed based on this camera posture. Then the image having been shotare transformed and pasted onto a map of the geographic informationsystem in conformity with this photographic frame, and the comparisonbetween the shot image and map is executed.

According to this third embodiment, a photographic frame is computedbased on posture of the camera acting as photographic means, therebyenabling to identify situations of the ground with higher precisionwhile confirming a positional relation between the shot image and themap.

Now, relations between the airframe 101 and the camera 102 are shown inFIGS. 5. On the assumption that the camera 102 is housed in the gimbal112, and the airframe 101 flies level, as shown in FIGS. 5 (b) and (c),an inclination of the camera 102 is outputted as an inclination of theairframe 101 with respect to a central axis (=tilt), and a rotationangle (pan) of the camera 102 is outputted as a rotation angle from atraveling direction of the airframe 101. That is, in the state of (b),the camera 102 faces right below so that an inclination is 0 degree. Inthe state of (c), an inclination θ of the camera 102 is shown to be aninclination with respect to the vertical plane.

The method of computing a photographic frame of the camera can beobtained with rotational movement and projection processing of arectangle (image frame) in 3D coordinates as a basis of computergraphics. Basically, a photographic frame of the camera is processed bytransformation between camera information and airframe information, anda graphic frame in the case of projecting this photographic frame to theground is computed, thereby enabling to obtain an intended image frame.A method of computing each coordinate in 3D coordinates is obtained byusing the following matrix calculation method.

1) Computing a Photographic Frame in the Reference State

First, as shown in FIG. 6 (a), positions of four points of an imageframe are computed as relative coordinates, letting a position of theairframe an origin. The photographic frame is computed into a referenceposition based on a focal length, angle of view and altitude of thecamera thereby obtaining coordinates of four points.

2) Computing Positions of Four Points After the Rotation About a Tilt ofthe Camera (Z-Axis)

As shown in FIG. 6 (b), a photographic frame is rotated about Z-axis inaccordance with a tilt angle θ of the camera. Coordinates after rotationare obtained by transformation with the following expression 1.$\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}{\cos\quad\theta} & {\sin\quad\theta} & 0 & 0 \\{{- \sin}\quad\theta} & {\cos\quad\theta} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 1} \right\rbrack\end{matrix}$3) Computing Positions of Four Points After the Rotation About anAzimuth of the Camera (y-axis)

As shown in FIG. 6 (c), a photographic frame is rotated about y-axis inaccordance with an azimuth θ of the camera. Coordinates after therotation are obtained by transformation with the following expression 2.$\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}{\cos\quad\theta} & 0 & {{- \sin}\quad\theta} & 0 \\0 & 1 & 0 & 0 \\{\sin\quad\theta} & 0 & {\cos\quad\theta} & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 2} \right\rbrack\end{matrix}$4) Calculating a Graphic Frame of Projecting the Image Frame AfterRotational Processing from an Origin (Airframe Position) to the GroundSurface (y-axis Altitude Point)

As shown in FIG. 6 (d), a projection plane (photographic frame) isobtained by projecting the photographic frame to the ground surface(y-axis altitude). Coordinates after projection are obtained bytransformation with the following expression 3. $\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & {1/d} \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 0\end{bmatrix}}} & \left\lbrack {{Expression}\quad 3} \right\rbrack\end{matrix}$

Generalized homogenous coordinate system [X, Y, Z, W] is obtained withthe following expression 4. In addition, d is a sea level altitude.[X Y Z W]=[x y z y/d]  [Expression 8]

Next, the expression 4 is divided by W′ (=y/d) and returned to be in 3D,resulting in the following expression 5. $\begin{matrix}{\begin{bmatrix}\frac{X}{W} & \frac{Y}{W} & \frac{Z}{W} & 1\end{bmatrix} = {\quad{\begin{bmatrix}{xp} & {yp} & {zp} & 1\end{bmatrix} = {\quad\begin{bmatrix}\frac{x}{y/d} & d & \frac{z}{y/d} & 1\end{bmatrix}}}}} & \left\lbrack {{Expression}\quad 5} \right\rbrack\end{matrix}$

Embodiment 4

According to this fourth embodiment, a current position of the airframe101 and an elevation angle and roll angle of the airframe 101 aremeasured, and a photographic frame of the ground having been shot fromon board is computed based on the elevation angle and roll angle. Thenan image having been shot is transformed and pasted onto a map of thegeographic information system in conformity with the photographic framethereof, and the comparison between the shot image and the map isexecuted. According to this fourth embodiment, a photographic frame iscomputed based on the positional information of the airframe 101 withrespect to the ground, thereby enabling to identify situations of theground with higher precision while confirming a positional relationbetween the shot image and map.

Now, as to relation between the airframe and the camera, let it beassumed that the camera 102 is fixed to the airframe 101 (that is, thegimbal is not used) as shown in FIG. 7. In the case where the airframe101 itself flies horizontally to the ground as shown in FIG. 7 (b), thecamera 102 faces right below so that inclination of the camera 102becomes 0 degree. In the case where the airframe 101 is inclined asshown in FIG. 7 (c), this inclination gives a posture of the camera 102and, therefore, a photographic frame of the camera is computed based onan elevation angle (pitch) and roll angle of the airframe 101.

1) Computing a Photographic Frame in the Reference State

As shown in FIG. 8 (a), positions of four points of an image frame arecomputed as relative coordinates, letting a position of the airframe anorigin. The photographic frame is computed into a reference positionbased on a focal length, angle of view, and altitude of the camera,thereby obtaining coordinates of four points.

2) Computing Positions of Four Points After the Rotation About a Roll ofthe Airframe (x-axis)

As shown in FIG. 8 (b), the photographic frame is rotated about x-axisin accordance with a roll angle θ of the airframe with the followingexpression. Coordinates after rotation are obtained by transformationwith the following expression 6. $\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}1 & 0 & 0 & 0 \\0 & {\cos\quad\theta} & {\sin\quad\theta} & 0 \\0 & {{- \sin}\quad\theta} & {\cos\quad\theta} & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 6} \right\rbrack\end{matrix}$3) Computing Positions of Four Points After the Rotation About a Pitchof the Airframe (z-axis)

As shown in FIG. 8 (c), the photographic frame is rotated about thez-axis in accordance with a pitch angle θ of the airframe. Coordinatesafter rotation are obtained by transformation with the followingexpression 7. $\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}{\cos\quad\theta} & {\sin\quad\theta} & 0 & 0 \\{{- \sin}\quad\theta} & {\cos\quad\theta} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 7} \right\rbrack\end{matrix}$4) Calculating a Graphic Frame of Projecting the Image Frame AfterRotation Processing From an Origin (Airframe Position) to a GroundSurface (y-axis Altitude Point)

As shown in FIG. 8 (d), a projection plane (photographic frame) isobtained by projecting the photographic frame to the ground surface(y-axis altitude). Coordinates after projection are obtained bytransformation with the following expression 8. $\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & {1/d} \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 0\end{bmatrix}}} & \left\lbrack {{Expression}\quad 8} \right\rbrack\end{matrix}$

Generalized homogenous coordinate system [X, Y, Z, W] is obtained withthe following expression 9.[X Y Z W]=[x y z y/d]  [Expression 9]

Next, the expression 9 is divided by W′ (=y/d) and returned to be in 3Dresulting in the following expression 10. $\begin{matrix}{\begin{bmatrix}\frac{X}{W} & \frac{Y}{W} & \frac{Z}{W} & 1\end{bmatrix} = {\quad{\begin{bmatrix}{xp} & {yp} & {zp} & 1\end{bmatrix} = {\quad\begin{bmatrix}\frac{x}{y/d} & d & \frac{z}{y/d} & 1\end{bmatrix}}}}} & \left\lbrack {{Expression}\quad 10} \right\rbrack\end{matrix}$

Embodiment 5

In this fifth embodiment, a current position of the airframe 101, arotation angle and inclination of the camera 102 with respect to theairframe, and further an elevation angle and roll angle of the airframe101 are measured, and a photographic frame of the ground having beenshot from on board is computed based on the information. Then an imagehaving been shot is transformed and pasted onto a map of the geographicinformation system in conformity with the photographic frame thereof,and the comparison between the image and the map is executed. Accordingto this fifth embodiment, a photographic frame is computed based onposture information of the camera and posture information of theairframe, thereby enabling to identify situations of the ground withhigher precision while confirming a positional relation between theimage and map.

Now, as to relation between the airframe 101 and the camera 102,supposing that the camera 102 is housed in the gimbal 112 as well as theairframe 101 flies in any posture as shown in FIG. 9, an inclination androtation angle of the camera 102 are outputted from the gimbal 112 asshown in FIG. 8 (b). Furthermore, an elevation angle and roll angle ofthe airframe 101 itself with respect to the ground are outputted fromthe gyro.

The method of computing a photographic frame of the camera can beobtained with rotational movement and projection processing of arectangle (image frame) in 3D coordinates as a basis of computergraphics. Basically, a photographic frame of the camera are processed bytransformation with camera information and airframe information, and agraphic frame in the case of projecting this photographic frame to theground is computed, thereby enabling to obtain an intended image frame.

The method of calculating each coordinate in 3D coordinates is obtainedby using the following matrix calculation method.

1) Computing a Photographic Frame in the Reference State

As shown in FIG. 10 (a), positions of four points of an image frame arecomputed as relative coordinates, letting a position of the airframe anorigin. A photographic frame is computed into a reference position basedon a focal length, angle of view, and altitude of the camera therebyobtaining coordinates of four points.

2) Computing Positions of Four Points After the Rotation About a Tilt ofthe Camera (Z-Axis)

As shown in FIG. 10 (b), transformation of rotating a shot image aboutZ-axis in accordance with a tilt angle θ of the camera is executed.Coordinates after rotation are obtained by transformation with thefollowing expression 11. $\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}{\cos\quad\theta} & {\sin\quad\theta} & 0 & 0 \\{{- \sin}\quad\theta} & {\cos\quad\theta} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 11} \right\rbrack\end{matrix}$3) Computing Positions of Four Points After the Rotation About anAzimuth of the Camera (y-axis)

As shown in FIG. 10 (c), transformation of rotating a photographic frameabout y-axis in accordance with an azimuth θ of the camera is executed.Coordinates after rotation are obtained by transformation with thefollowing expression 12. $\begin{matrix}{\begin{bmatrix}x^{\prime} & y^{\prime} & z^{\prime} & 1\end{bmatrix} = {\begin{bmatrix}x & y & z & 1\end{bmatrix}\begin{bmatrix}{\cos\quad\theta} & 0 & {{- \sin}\quad\theta} & 0 \\0 & 1 & 0 & 0 \\{\sin\quad\theta} & 0 & {\cos\quad\theta} & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 12} \right\rbrack\end{matrix}$4) Computing Positions of Four Points After the Rotation About a Roll ofthe Airframe (x-axis)

As shown in FIG. 10 (d), transformation of rotating a photographic frameabout x-axis in accordance with a roll angle θ of the airframe isexecuted. Coordinates after rotation are obtained by transformation withthe following expression 13. $\begin{matrix}{\left\lbrack {x^{\prime}\quad y^{\prime}\quad z^{\prime}\quad 1} \right\rbrack = {\left\lbrack {x\quad y\quad z\quad 1} \right\rbrack\begin{bmatrix}1 & 0 & 0 & 0 \\0 & {\cos\quad\theta} & {\sin\quad\theta} & 0 \\0 & {{- \sin}\quad\theta} & {\cos\quad\theta} & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 13} \right\rbrack\end{matrix}$5) Computing Positions of Four Points After the Rotation About a Pitchof the Airframe (z-axis)

As shown in FIG. 10 (e), transformation of rotating a photographic frameabout z-axis in accordance with a pitch angle θ of the airframe isexecuted. Coordinates after rotation are obtained by transformation withthe following expression 14. $\begin{matrix}{\left\lbrack {x^{\prime}\quad y^{\prime}\quad z^{\prime}\quad 1} \right\rbrack = {\left\lbrack {x\quad y\quad z\quad 1} \right\rbrack\begin{bmatrix}{\cos\quad\theta} & {\sin\quad\theta} & 0 & 0 \\{{- \sin}\quad\theta} & {\cos\quad\theta} & 0 & 0 \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix}}} & \left\lbrack {{Expression}\quad 14} \right\rbrack\end{matrix}$6) Calculating a Graphic Frame of Projecting the Image Frame AfterRotational Processing from an Origin (Airframe Position) to a GroundSurface (y-axis Altitude Point)

As shown in FIG. 10 (f), a projection plane (photographic frame) isobtained by projecting the photographic frame to the ground surface(y-axis altitude). Coordinates after projection are obtained bytransformation with the following expression 15. $\begin{matrix}{\left\lbrack {x^{\prime}\quad y^{\prime}\quad z^{\prime}\quad 1} \right\rbrack = {\left\lbrack {x\quad y\quad z\quad 1} \right\rbrack\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & {1/d} \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 0\end{bmatrix}}} & \left\lbrack {{Expression}\quad 15} \right\rbrack\end{matrix}$7) Generalized Homogenous Coordinate System [X, Y, Z, W] is Obtainedwith the Following Expression 16.[X Y Z W]=[x y z y/d]  [Expression16]8) Next, the Expression 16 is Divided by W′(=y/d) and Returned to be in3D Resulting in the Following Expression 17. $\begin{matrix}\begin{matrix}{\left\lbrack {\frac{X}{W}\quad\frac{Y}{W}\quad\frac{Z}{W}\quad 1} \right\rbrack = \left\lbrack {{xp}\quad{yp}\quad{zp}\quad 1} \right\rbrack} \\{= \left\lbrack {\frac{x}{y/d}\quad d\quad\frac{z}{y/d}\quad 1} \right\rbrack}\end{matrix} & \left\lbrack {{Expression}\quad 17} \right\rbrack\end{matrix}$

Embodiment 6

In this sixth embodiment, a current position of the airframe 101, arotation angle and inclination of the camera 102 with respect to theairframe, and further an elevation angle and roll angle of the airframe101 are measured, and a photographic frame of the ground having beenshot from on board is computed into a map of the geographic informationsystem. In computing processing of four points of this photographicframe, topographic altitude data is utilized, and a flight position ofthe airframe 101 is compensated to compute the photographic frame. Thenan image having been shot is transformed in conformity with thephotographic frame thereof and pasted onto a map of the geographicinformation system, and the comparison between the shot image and map isexecuted.

According to this sixth embodiment, the compensation is executed withaltitude topographic information of the surface ground using informationabout a position and altitude of the airframe, airframe postureinformation and posture information of the camera, and a photographicframe is computed, thereby enabling to identify with higher precisionsituations of the ground while confirming a positional relation betweenthe image and the map.

In the foregoing fifth embodiment, a sea level altitude obtained fromthe GPS apparatus is employed as an altitude of the airframe incomputing processing of a photographic frame onto the ground surfaceafter rotation: whereas, in this sixth embodiment, as shown in FIG. 11,a ground surface altitude (relative altitude d=sea level altitude−groundsurface altitude) at a shooting point is employed as an altitude of theairframe utilizing a topographic altitude information of the groundsurface. In this manner, computing four points of a photographic frameis executed.

1) Calculating a Graphic Frame of Projecting an Image Frame AfterRotational Processing from an Origin (Airframe Position) to the GroundSurface (y-axis Altitude Point)

A projection plane is obtained by projecting the photographic frame tothe ground surface (y-axis altitude) Coordinates after projection areobtained by transformation with the following expression 18.$\begin{matrix}{\left\lbrack {x^{\prime}\quad y^{\prime}\quad z^{\prime}\quad 1} \right\rbrack = {\left\lbrack {x\quad y\quad z\quad 1} \right\rbrack\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & {1/d} \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 0\end{bmatrix}}} & \left\lbrack {{Expression}\quad 18} \right\rbrack\end{matrix}$

Generalized homogenous coordinate system [X, Y, Z, W] is obtained withthe following expression 19.[X Y Z W]=[x y z y/d]  [Expression 19]

Next, the expression 19is divided by W′(=y/d) and restored to be in 3Dresulting in the following expression 20. $\begin{matrix}\begin{matrix}{\left\lbrack {\frac{X}{W}\quad\frac{Y}{W}\quad\frac{Z}{W}\quad 1} \right\rbrack = \left\lbrack {{xp}\quad{yp}\quad{zp}\quad 1} \right\rbrack} \\{= \left\lbrack {\frac{x}{y/d}\quad d\quad\frac{z}{y/d}\quad 1} \right\rbrack}\end{matrix} & \left\lbrack {{Expression}\quad 20} \right\rbrack\end{matrix}$

A relative altitude d, which is used herein, is obtained by subtractinga topographic altitude at a target point of land from an absolutealtitude from the horizon, which is obtained from the GPS apparatus.Further this relative altitude from the camera is utilized, therebyenabling to compute with higher precision the position of a photographicframe.

Embodiment 7.

In this seventh embodiment, at the time of measuring a current positionof the airframe 101, computing a photographic frame of the ground havingbeen shot from on board on a map of the geographic information system,transforming an image having been shot in conformity with thephotographic frame thereof and pasting it, and executing the comparisonbetween the shot image and map, plural pieces of shot images to bepasted in succession on the map are displayed being pasted continuouslyonto the map of the geographic information system. Then a target pointof land is specified with the pasted images on the map. In theprocessing of pasting plural pieces of shot images onto a map of thegeographic information system, the layout is performed in accordancewith the computed photographic frames, a joint state of overlap part ofeach shot image is confirmed, and the images are moved so that overlapcondition of the images may be of the largest extent to execute thepositional compensation. Then, the shot images are transformed inconformity with the photographic frames on the map of the geographicinformation system with the use of the compensation values, and pasteprocessing is performed.

Procedures thereof are shown in FIG. 12. For example, two pieces of shotimages 1(A) and 2(B), which are taken as the airframe 101 travels, areoverlapped, and an overlap part (internal part of a solid frame C of thedrawing) is detected. Then the images A and B are moved relatively sothat the overlap condition of the images maybe of the largest extent, apositional compensation value at the time of joining is obtained, thepositional compensation D is executed, and the images A and B arejoined. The positional compensation is carried out in image joiningcompensation 215 of FIG. 2.

According to this seventh embodiment, plural pieces of continuous imagesprovide a more precise joining, thereby enabling to identify situationsof the ground while confirming situations of a wider range of groundsurface.

Embodiment 8

In this eighth embodiment, a current position of the airframe 101, amounting angle and inclination of the camera 102 with respect to theairframe, and further an elevation angle and roll angle of the airframe101 are measured. Then a photographic frame of the ground having beenshot from on board is computed, the image is transformed in conformitywith the photographic frame thereof to be pasted onto a map of thegeographic information system, and the comparison between the sot imageand map is executed.

In the case of executing this processing, it comes to be important thatvarious information, which are transmitted from the on-board system 100,are received at the ground system 200 fully in synchronization. Toachieve this synchronization, it is necessary to adjust a processingtime period such as processing time period of flight positionmeasurement means, processing time period of posture measurement meanswith the gimbal of camera and a processing time period of imagetransmission, and to transmit them in synchronization with the shotimage. To actualize this synchronization, a buffer is provided in theconstruction of FIG. 1, and image signals of the camera on board aretemporarily stored with storage means 113 in this buffer and transmittedto the ground system 200 in synchronization with the forgoinginformation after the delay of a computing time period for airframepositional detection by, e.g., GPS.

This relation is described with reference to FIG. 13. A time period T isrequired for the airframe 101 to receive a GPS signal and detect anairframe position, and during this time period the airframe 101 travelsfrom a position P1 to a position P2. Therefore at the instant ofcompleting a positional detection of the airframe, a region, which thecamera 102 shoots, comes to be a region apart from the region, which thecamera 102 has shot at the position P1, by a distance R resulting inoccurrence of error.

FIG. 13 (b) is a time-chart showing procedures of correcting this error.An image signal is temporarily stored in the buffer during a GPScomputing time period T from a GPS observation point t1 for airframepositional detection, and the image signal having been storedtemporarily is transmitted together with airframe position, airframeposture, camera information and the like at the instant of t2.

According to this eighth embodiment, a photographic frame is computedbased on mounting information of the photographic device, therebyenabling to identify with higher precision situations of the groundwhile confirming a positional relation between the image and map.

Further, according to each of the foregoing embodiments, an image frameis computed, thereafter a shot image is transformed in conformity withthis image frame, and this transformed image is superposed and pastedonto a map. However, it is preferable that a photographic area on themap corresponding to a shot image, which photographic means has taken,is merely obtained, and the shot image is superposed on this area of themap to be displayed.

Furthermore, according to each of the foregoing embodiments, mapprocessing is executed at the ground system based on information to betransmitted from the on-board system. However, this map processing isnot limited thereto, and it is preferable that the on-board system isprovided with a monitor such as display, the map processing is executedat the on-board system, and the processed map is displayed on themonitor of the on-board system; or that information having beenprocessed is transmitted to the ground system, and displayed at theground system.

Embodiment 9

According to this ninth embodiment, so-called land marks, for example, across point or station or a large building corner that show remarkablepoints on the map are extracted from a shot image; and the correspondinglandmark is extracted from a region corresponding to the photographicarea on the map. Further, parameters for image frame computing(hereinafter, showing information of airframe position, airframe postureand camera posture, and camera set information for use in computing aphotographic frame, being a photographic area of the camera on theground surface) are adjusted so that the landmarks of the image and themap may be in coincidence, whereby the image is transformed anddisplayed being superposed on a GIS screen with high precision.

Hereinafter, descriptions are made referring to the drawings. FIG. 14 isa block diagram showing the ninth embodiment. Additionally, in FIG. 14,diagrammatic representations of the antenna 103, multiplex modulator109, signal conversion means 110, tracking means 111, temporary storagemeans 113, transmission means 104, receiving means 201, tracking means202, signal conversion means 203, and multiplex demodulator 204 areomitted. FIG. 15 is a function explanatory diagram for explaining mapprocessing means.

With reference to FIG. 14, current positional information is obtainedwith airframe position measurement means 108 such as GPS apparatus thatis mounted on a flight vehicle (=airframe) such as helicopter, and theairframe positional measurement is performed. Furthermore, the airframe101 comprises, e.g., gyro, and posture, i.e., an elevation angle(=pitch) and roll angle are measured with this airframe posturemeasurement means 107. Photographic means 105, being the camera 102mounted on the airframe 101 takes a shot of the ground, and outputsimage signals thereof as well as outputs camera information such as zoomof the camera. The camera 102 is attached to, e.g., gimbal, and arotation angle (=pan) and inclination (=tilt) of the camera is measuredwith this camera posture measurement means 106.

Outputs from these airframe position measurement means 108, airframeposture measurement means 107, photographic means 105, and cameraposture measurement means 106 are inputted to signal processing means205 and signal-processed respectively. Image signals of camera shootingare converted to a moving image data 207 and a still image data 208.Outputs from the signal processing means 205 and a two-dimensional mapdata 209 are inputted to map processing means 226, and the mapprocessing is executed.

The map processing means 226 includes functions shown in FIG. 15. In themap processing means 226, as shown in FIG. 15, the processing isexecuted based on a moving image data 207 and a still image data 208,being image signals, and information signals of airframe position,airframe posture, and camera posture, and a two-dimensional map data 209of the geographic information system.

In the map processing means 226, first image frame computing 212 isexecuted, in which a shooting position in the air is specifiedthree-dimensionally, and a photographic area (=photographic frame) ofthe ground surface having been shot is obtained by computation based onposture of the camera and airframe with respect to the ground surface.Then, landmark extraction 220 is executed to an extent corresponding tothe photographic area and its vicinity on a map of the geographicinformation system, and landmark extraction 221 is executed also from astill image data 208. Landmark comparison 222 for causing theselandmarks in coincidence is executed. Image transformation compensation223 is executed based on a result of the landmark comparison 222, and asuperposed display position of a shot image onto the map is compensated.Thereafter, superposition 214 of the image on the map of the geographicinformation system is executed. Finally, this superposed picture isdisplayed on a monitor with monitor display means 211 such as CRT.

Now, operations are described based on a flowchart of FIG. 16. First, anairframe position, being an output from airframe position measurementmeans 108, a pitch elevation angle and roll angle, being an output fromairframe posture measurement means 107, a pan and tilt, being an outputfrom camera posture measurement means 106, a zoom of the camera 102,being an output from photographic means 105, a still image data 208obtained with signal processing means 205, and a two-dimensional mapdata 209 are read in as input data respectively (S2). Next, the imageframe computing 212 is executed using an airframe position, pitchelevation angle, roll angle, and a pan, tilt and zoom of the camera asparameters (S22). Subsequently, the landmark extraction on the map of ageographic information system is executed about a region correspondingto a photographic frame obtained by the image frame computing 212 (S23).In the case where any landmark is extracted in S23, the correspondinglandmark is extracted from a still image data 208 (S24) (S25).

In the case where the landmark is extracted also from an image in S25,the corresponding landmarks that are obtained in S23 and S25 arecompared with each other, and parameter (for example, pan tilt) valueshaving been used in the image frame computing of S22 are compensated sothat these landmarks are in coincidence (S26) (S27) (S28). Further, thephotographic frame is computed again based on the compensation value ofparameters having been obtained in S28, and a still image data 208 istransformed in conformity with this photographic frame and displayedbeing superposed on a map of the geographic information system (S29)(S30) (S31).

In the case where any landmark is not extracted in S23 or S25, a stillimage data 208 is transformed in conformity with a photographic frameobtained in S22, and displayed being superposed on a map of thegeographic information system (S24) (S26) (S30) (S31). FIG. 17 shows apitch elevation angle, rotation angle, and a pan and tile of the camera,being angle parameters for use in the image frame computation 212.

For the computing method of a photographic frame, the above-describedmethod is employed. A photographic frame in the reference state isrotationally-processed with each angle parameter, and thereafterprojected onto the ground surface, whereby a photographic area of thecamera on the ground surface, that is, a photographic frame is obtained.As shown in FIG. 18, when x-axis is laid in airframe travelingdirection, z-axis is laid in vertically upward direction with respect tothe ground surface, and y-axis is laid so as to be vertical to thesex-axis and z-axis, letting an airframe position an origin, the specificcomputation is as follows:

-   Computing a photographic frame in the reference state-   Rotation about y-axis based on a tilt of the camera-   Rotation about z-axis based on a pan of the cameral-   Rotation about x-axis based on a roll angle of the airframe-   Rotation about y-axis based on a pitch elevation angle of the    airframe-   Projection onto the ground surface (horizontal surface of absolute    altitude (=sea level altitude) 0)

FIG. 18(a) shows the state in which a photographic frame 42 is computedinto the reference state. FIG. 18(b) shows the state in which thephotographic frame 42 of the reference state is rotationally processedwith each angle parameter, and thereafter projected onto the groundsurface.

The method of compensating a pan and tilt of the camera is now describedreferring to FIG. 19. When letting an airframe altitude h, a measuredvalue of tilt θ, a measured value of pan φ, landmark coordinates on animage (x, y) and landmark coordinates on the map (x₀, y₀), values oftilt and pan after the compensation θ₀, θ₀ can be obtained by workingout the following expression 21. $\begin{matrix}\left\{ \begin{matrix}{{{h \cdot \tan}\quad{\theta_{0} \cdot \cos}\quad\phi_{0}} = x_{0}} \\{{{h \cdot \tan}\quad{\theta_{0} \cdot \sin}\quad\phi_{0}} = y_{0}}\end{matrix} \right. & \left\lbrack {{Expression}\quad 21} \right\rbrack\end{matrix}$where: landmark coordinates (x₀, y₀) on the map to compare herein arecoordinates after the following transformation.

-   Counter-rotation about y-axis based on a pitch elevation angle of    the airframe-   Counter-rotation about x-axis based on a roll angle of the airframe-   Projection onto the ground surface (horizontal surface of absolute    altitude (=sea level altitude) 0)

FIG. 20 (a) is a picture of a photographic frame 42 and a shot image 43being superposed onto a map 41 of the geographic information systemwithout compensation according to the invention. FIG. 20(b) is a pictureafter being subjected to the compensation according to the invention,showing a photographic fame 42 and a shot image 43 being superposed onthe map 41 of a geographic information system. Numeral 44 indicates anairframe position (camera position). By the processing with mapprocessing means 226 including the above-described compensationprocessing, an image and a map are in coincidence at all points, asshown in FIG. 20(b), thus enabling to carry out a superposed displaywith high precision, and to understand situations of the ground surfacehaving been shot more easily and rapidly.

According to this ninth embodiment, not only it is possible to correctmeasuring error of various measurement devices that measure eachparameter; but also it becomes possible to correct error having occurreddue to lag in timing between shooting and data-getting of camera postureinformation (pan tilt) in the case of superposing and displaying animage having been shot during operation of a camera that is mounted onthe airframe on the map.

Embodiment 10

This tenth embodiment is a method of making the parameter adjustment ofthe above-mentioned ninth embodiment not by the compensation of pan andtilt but by the compensation of posture information (roll and pitch) ofthe airframe, thereby compensating position of a photographic frame. Thecompensation of roll and pitch is executed with the followingcomputation.

When letting landmark coordinates on an image at the time of completingthe rotational processing with a tilt and pan (x₁, y₁, z₁), landmarkcoordinates (x₂, y₂, z₂) at the time of having executed the rotationalprocessing with a roll θ and pitch φ is obtained with the followingexpression 22. $\begin{matrix}{\left( {x_{2}\quad y_{2}\quad z_{2}} \right) = {\left( {x_{1}\quad y_{1}\quad z_{1}} \right)\begin{pmatrix}{\cos\quad\theta} & 0 & {\sin\quad\theta} \\0 & 1 & 0 \\{{- \sin}\quad\theta} & 0 & {\cos\quad\theta}\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\quad\phi} & {\sin\quad\phi} \\0 & {{- \sin}\quad\phi} & {\cos\quad\phi}\end{pmatrix}}} & \left\lbrack {{Expression}\quad 22} \right\rbrack\end{matrix}$Further, when performing the projection onto the ground surface,landmark coordinates (x, y, z) are obtained with the followingexpression 23. $\begin{matrix}{\left( {x\quad y\quad z} \right) = {\left( {x_{2}\quad y_{2}\quad z_{2}} \right) \cdot \frac{h}{z_{2}}}} & \left\lbrack {{Expression}\quad 23} \right\rbrack\end{matrix}$Herein, an alphabet h is an airframe altitude, and θ, φ satisfying thefollowing expression 24 $\begin{matrix}\left\{ \begin{matrix}{{x\left( {\theta,\phi} \right)} = x_{0}} \\{{y\left( {\theta,\phi} \right)} = y_{0}}\end{matrix} \right. & \left\lbrack {{Expression}\quad 24} \right\rbrack\end{matrix}$when letting landmark coordinates on the map (x₀, y₀), are roll θ₀,pitch φ₀ after the compensation.

According to this tenth embodiment, since the camera is fixedly attachedto the airframe, and mounted so that an angle of pan and tilt is notvaried, the compensation of parameters in the more real state can bemade by correcting posture information of the airframe that is a rolland pitch even in the case where the compensation with a pan an tilt isin effective, thus enabling to carry out a more precisely superposeddisplay. As a result, it is possible to understand situations of theground surface having been shot more easily and rapidly.

Embodiment 11

According to this eleventh embodiment, 2 points of landmarks areextracted, and the altitude compensation of the airframe is made with adistance between these 2 points. In the case where 2 points of landmarksare extracted in S23 of the ninth embodiment (FIG. 16), thecorresponding 2 points of landmarks are likewise extracted also from astill image data (S24) (S25).

In the case where the corresponding landmarks are extracted also from animage in S25, the landmarks having been obtained in S23 and S25 arecompared, and an airframe altitude is compensated so that a distancebetween 2 points of landmarks on the image and a distance between 2points of landmarks on the GIS map (in this case, since an airframealtitude is obtained as an absolute altitude from the sea level with theGPS apparatus, a relative altitude from the ground surface will beobtained by this altitude compensation) (S27) (S28).

Further, a photographic frame is computed again based on thecompensation values of parameters that are obtained in S28, a stillimage data 208 is transformed in conformity with this photographic frameand displayed being superposed on a map of a geographic informationsystem (S29) (S30) (S31)

As seen from FIG. 21(b), an altitude (relative altitude) h′ having beencompensated with a distance between landmarks according to the inventionis obtained with the expression of(relative altitude)=(absolute altitude)×(distance between 2 points oflandmarks on a map)/(distance between 2 points of landmarks on animage),letting an absolute altitude of the airframe h. In the drawing, E is adistance on the map and F is a distance on the image.

By the processing with map processing means 226 including theabove-described compensation processing, a shot image with respect to apoint of land of which ground surface is higher than the sea level canbe displayed being superposed with high precision, thereby enabling tounderstand situations of the ground surface having been shot more easilyand rapidly.

Embodiment 12

This twelfth embodiment is intended to make it possible that a shotimage and map are displayed being superposed with higher precision bycompensating parameters in accordance with the number of landmarks. Inthe case where 2 points of landmarks are extracted in S22 of theforegoing ninth embodiment (FIG. 16), the corresponding 2 points oflandmarks are likewise extracted also from a still image data 208 (S24)(S25). In the case where the landmarks are extracted also from an imagein S25, the corresponding landmarks obtained in S23 and S25 arecompared.

First, parameter (pan and tilt) values having been used in the imageframe computing of S22 are compensated so that the first correspondinglandmarks are in coincidence, and next airframe posture parameter (rolland pitch) values are compensated so that a difference between thesecond corresponding landmarks are corrected (S27) (S28). Further, aphotographic frame is computed again based on the compensation values ofeach parameter obtained in S28, and a still image data 208 istransformed in conformity with this photographic frame and displayedbeing superposed on the map of the geographic information system (S29)(S30) (S31).

FIG. 22 is a diagram explaining this compensation, and in which blackcircle marks indicate landmarks on the map and filled triangle marksindicate landmarks on the image. FIG. 22(a) shows the state in which ashot image is displayed being superposed on the GIS map; FIG. 22(b)shows the state after the altitude compensation according to theforegoing eleventh embodiment has been executed; FIG. 22 (c) shows thestate after the pan and tilt compensation has been executed thereafter;and FIG. 22(d) shows the state after the roll and pitch compensation hasbeen further executed.

According to this twelfth embodiment, even in the case where it isdifficult that a shot image and a map are displayed being superposedwith high precision over the entire photographic area by the adjustmentof parameters with only 1 point of landmarks in coincidence, thesuperposed display with higher precision can be achieved by using 2points of landmarks, thereby enabling to understand situations of theground surface having been shot more easily and rapidly.

Embodiment 13

According to this thirteenth embodiment, in the case where not less than3 points of landmarks are extracted, parameter compensation valuesbetween all the two points, and an average value thereof is used as aparameter compensation value. In the case where a plurality of landmarksof not less than 2 points are extracted in S23 of the ninth embodiment(FIG. 16), the corresponding plural landmarks of not less than 2 pointsare likewise extracted also from a still image data 208 (S24) (S25).

In the case where landmarks are extracted also from an image in S25, thecorresponding 2 points are picked up from the landmarks having beenobtained in S23 and S25, and respective comparisons are executed,thereby obtaining a compensation value of parameters. This processing isexecuted as to all selections of 2 points of landmarks, whereby aplurality of parameter compensation values are obtained, an average ofthese compensation values as to respective parameters is obtained, andthese average values are used as a compensation value of respectiveparameters (S27) (S28). Further, a photographic frame is computed againbased on the compensation value of parameters having been obtained inS28, and a still image data 208 is transformed in conformity with thisphotographic frame and displayed being superposed on a map of thegeographic information system (S29) (S30) (S31).

By the map processing including the mentioned compensation processing,as compared with the case of compensating the superposed display of animage and map based on positions of 1 or 2 points of landmarks, it isnow possible to achieve the superposed display with higher precision,thereby enabling to understand situation of the ground surface havingbeen shot more easily and rapidly.

Embodiment 14

This fourteenth embodiment relates to superposed display processing ontothe map in the case where plural pieces of images are shot continuouslyin cycles of a predetermined time period and a series of plural imagesare provided as a still image data. The extraction processing oflandmarks is carried out with respect to an obtained still image. As aresult, supposing that landmarks are extracted, the compensation isexecuted by the comparison with the GIS map. However, landmarks cannotalways be extracted from all the still images. In the live displayprocessing of performing the superposed display while taking a shot, itis difficult to instantly execute image processing to extract landmarksand perform the compensation with respect to all shot images due toprocessing time period.

Therefore, as for the superposed display on the map of a still image inwhich landmarks are not extracted, image fame computation is executedagain based on compensation values at the time of the last compensation,an image is transformed in conformity with the photographic frame havingbeen obtained and displayed being superposed on a map of the geographicinformation system, thereby achieving improvement in precision of thesuperposed position.

This processing corresponds to S24, S26, S32, S33, S31 of FIG. 23. Inthe case where any corresponding landmark is extracted in S24, the sameprocessing as in the ninth embodiment is executed. FIG. 24 shows amonitor display screen according to this method. Numeral 41 designates amap; numeral 44 designates an airframe position (camera position); andnumeral 45 designates a flight path of the airframe. Images having beenshot with the camera along the flight path 45 are sampled with apredetermined timing, subjected to the superposed positionalcompensation respectively, and there after displayed being superposed ona map 41 of the geographic information system. Numerals 43 a to 43 gdesignate pasted images. Numeral 42 designates a photographic frame ofthe latest image 43 g.

According to this fourteenth embodiment, even in the case where nolandmarks are extracted, it is possible to compensate superposed displaypositions, thereby enabling to carry out the superposed display withhigh precision, as well as enabling to understand situations of a widerange of the ground surface having been shot more easily and rapidly.

Embodiment 15

The fifteenth embodiment relates to superposed display processing ontothe map in the case where plural pieces of images are shot continuouslyin cycles of a predetermined time period and a series of plural imagesare provided as a still image data. As for images having beencontinuously shot, there are some images that are subjected to thesuperposed positional compensation by the comparison of landmarks, andother images with which the superposed positional compensation by thecomparison cannot be performed.

In this case, at the time of real flight, as shown in the foregoingfourteenth embodiment, the last compensation values continue to be useduntil the next landmark is extracted. However, in the processing ofsuperposed display of an image on a map with the use of any image ofpast flight, a processing time period for positional compensation canafford to be spent as compared with the case of live flight.Accordingly, in the case where the image of past flight is displayedbeing superposed on the map, as shown in FIG. 25, compensation values ofeach parameter that are obtained at a point of land where the nextlandmarks are extracted are applied, going back to the halfway pointbetween the point of having executed the compensation by landmarkcomparison last and the current point.

With reference to FIG. 25, a gray square indicates a landmark extractionimage, and a white square shows an image from which no landmark isextracted. Further, an arrow shows that superposed positionalcompensation values are utilized from an image from which landmarks areextracted and with which the superposed positional compensation has beenexecuted, and a superposed position is compensated. According to thisfifteenth embodiment, an overlap state between images in the case whereany compensation by comparison of landmarks cannot be executed isimproved as shown in FIG. 25.

FIG. 25 (a) shows the case where this fifteenth embodiment is notapplied, and FIG. 25(b) shows the case where this fifteenth embodimentis applied. A shot image with which the superposed display positionalcompensation of image by the comparison of landmarks can be executed istaken as a base point, and the layout of images are adjusted back andforth so as to maximize the rate of coincidence of overlap parts ofimages toward the halfway point between the shot images with which thesuperposed display compensation is executed, whereby the images havingbeen continuously shot can be displayed being superposed on the GIS mapwith higher precision.

According to the fifteenth embodiment, in the processing of superposingand displaying the images of past flight on the GIS map, it is possibleto execute the compensation of superposed display positions even in thecase where no landmark is extracted. Furthermore, the overlappingcondition between the images is not segmented with the image from whicha landmark is extracted, thus enabling to carry out the superposeddisplay in more smooth succession with high precision, as well asenabling to understand situations of a wide range of the ground surfacehaving been shot more easily and rapidly.

Embodiment 16

According to this sixteenth embodiment, an altitude compensation data ofa shot image to be extracted from flight images of the past is linked toa position and registered, whereby altitude compensation of a shot pointof land is executed even in the case where landmarks cannot be extractedfrom a shot image.

In the case where the altitude compensation processing can be executedwith the coincidence of landmarks, an altitude compensation valueobtained as a difference between absolute altitude and a relativealtitude is registered and managed at a shot point of land as analtitude compensation value of this point, whereby, this altitudecompensation value can be utilized at any time. Further, in the casewhere the airframe flies at a point of land close to the foregoing pointand from the next flight on, the altitude compensation can be executedeven at the time of live flight when a processing time period islimited, or even in the case where not less than 2 points ofcorresponding landmarks cannot be extracted in a still image and a map.

FIG. 26 shows a state in which still images having been continuouslyshot are displayed being superposed on the GIS map. Explained in thisdrawing is the case where 2 points of landmarks are extracted from thelast one piece of image 51 and the intermediate one piece of image 52,and a compensation value of altitude can be obtained.

Not less than 2 points of landmarks are in coincidence with the image 51and the image 52, thereby enabling to obtain a compensation value ofaltitude. When letting these compensation values 61 and 62 respectively,the altitude compensation values 61 and 62 at points on the map areregistered as symbols. With respect to an image from which not less than2 points of landmarks cannot be extracted, an altitude compensationvalue at this point of land is provided, thus executing the compensationof error due to not only a mounting angle of the camera but also analtitude of the ground surface, thereby enabling to superpose anddisplay images having been continuously shot on the GIS map with higherprecision.

According to the sixteenth embodiment, by registration of an altitudecompensation data having been extracted from the images of past flightat a point on the map, it is possible to carry out the altitudecompensation with respect to an image from which not less than 2 pointsof landmarks cannot be extracted, thereby enabling the superposeddisplay with higher precision.

INDUSTRIAL APPLICABILITY

The present invention is applicable to an image display apparatus takinga shot of situations on the ground from on board such as helicopter inthe case where natural disaster such as earthquake or fire occurs orwhere human disaster such as explosion or serious accident occur.

1-20. (canceled)
 21. An image display apparatus of image processing anddisplaying a shot image of the ground surface having been taken withphotographic equipment that is mounted on an airframe in the air, theimage display apparatus comprising: an image frame computing means inwhich a shooting position in the air is specified three-dimensionallybased on posture of said airframe and said photographic equipment withrespect to said ground surface and a photographic area on the groundsurface having been shot is obtained by computation; an imagetransformation means in which a shot image is transformed in conformitywith said photographic area; a superposing means in which saidtransformed image is superposed on a map of a geographic informationsystem; and a monitor display means for displaying said superposed map.22. An image display apparatus of image processing and displaying a shotimage of the ground surface having been taken with photographicequipment that is mounted on an airframe in the air, the image displayapparatus comprising: an image frame computing means in which a shootingposition in the air is specified three-dimensionally based on posture ofsaid airframe and said photographic equipment with respect to saidground surface, and each of a plurality of photographic areas on theground surface having been continuously shot is obtained by computation;an image transformation means in which each of shot images istransformed in conformity with said each of the photographic areas; asuperposing means in which said plurality of transformed shot images aresuperposed on a map of a geographic information system; and a monitordisplay means for displaying said superposed map.
 23. The image displayapparatus according to claim 22, further comprising an image joining andcompensating means in which a plurality of shot images to be superposedare partially overlapped with each other, and the shot images are movedand compensated so that an overlapping at the overlap part may be of thelargest extent and thereafter are joined.
 24. The image displayapparatus according to claim 22, wherein said plurality of shot imagesto be superposed are obtained by sampling the images having beencontinuously shot in cycles of a predetermined time period.
 25. Theimage display apparatus according to claim 21, wherein said image framecomputing means obtains a photographic area on the ground surface havingbeen shot by computation based on an inclination and a rotation angle ofsaid photographic equipment with respect to said airframe.
 26. The imagedisplay apparatus according to claim 21, wherein said image framecomputing means obtains a photographic area on the ground surface havingbeen shot by computation based on an inclination and a roll angle ofsaid airframe with respect to the ground surface.
 27. The image displayapparatus according to claim 21, wherein said image frame computingmeans obtains a photographic area on the ground surface having been shotby computation based on an inclination and rotation angle of saidphotographic equipment with respect to said airframe, and an inclinationand roll angle of said airframe with respect to the ground surface. 28.The image display apparatus according to claim 27, wherein said imageframe computing means obtains a photographic area on the ground surfaceby computation, thereafter obtains an altitude of the ground surface ofsaid photographic area by utilizing a three-dimensional topographic dataincluding altitude information as to undulation of the ground surfacewhich data has been preliminarily prepared, computes an altitude ofshooting point as a relative altitude obtained by subtracting analtitude of the ground surface from an absolute altitude of theairframe; the image transformation means transforms a shot image inconformity with said photographic area; and the superposing meansdisplays the transformed image so as to be superposed on a map of thegeographic information system.
 29. An image display apparatus for takinga shot of the ground surface with photographic equipment that is mountedon an airframe in the air, and to identify situations existing on saidground surface by comparison between a shot image and a map; wherein ashooting position in the air is specified three-dimensionally based onposture of said airframe and said photographic equipment with respect tosaid ground surface, and signals of said airframe positionalinformation, camera information, and airframe information aretransmitted in synchronization with signals of an image having beenshot; and a photographic area on the ground surface having been shot isobtained by computation on the receiving side, and a shot image istransformed in conformity with said photographic area and thereafterdisplayed being superposed on a map of a geographic information system.30. An image display method of image processing and displaying a shotimage of the ground surface having been taken with photographicequipment that is mounted on an airframe in the air, wherein a shootingposition in the air is specified three-dimensionally based on posture ofsaid airframe and said photographic equipment with respect to saidground surface and, a photographic area of at least one image of theground surface having been shot is obtained by computation; a shot imageis transformed in conformity with said photographic area; thereafter thetransformed shot image is displayed being superposed on a map of ageographic information system, and the shot image having been superposedon the map can be erased leaving only a photographic area frame.
 31. Theimage display method according to claim 30, wherein landmarks areextracted from a map of said geographic information system and said shotimage respectively, and the corresponding landmarks are compared,whereby a parameter for use in computing a photographic area of theground surface having been shot is compensated, and a shot image isdisplayed being superposed with high precision on a map of thegeographic information system.
 32. The image display method according toclaim 31, wherein a parameter to be compensated is changed in accordancewith the number of landmarks having been extracted.
 33. The imagedisplay method according to claim 31, wherein an inclination and arotation angle of said photographic equipment with respect to saidairframe are compensated based on the landmark having been extracted,and a photographic area on the ground surface having been shot iscomputed.
 34. The image display method according to claim 31, wherein aninclination and a roll angle of said airframe with respect to the groundsurface are compensated based on the landmark having been extracted, anda photographic area on the ground surface having been shot is computed.35. The image display method according to claim 31, wherein 2 points oflandmarks having been extracted are used, an altitude of said airframeis compensated based on a distance between two points, and aphotographic area of the ground surface having been shot is computed.36. The image display method according to claim 31, wherein an averagevalue of parameter compensation values between 2 points of each landmarkis used in the case of not less than 3 landmarks having been extracted,and a photographic area of the ground surface having been shot iscomputed.
 37. The image display method according to claim 31, wherein inthe case of absence of the corresponding landmarks at the time ofextracting landmarks from a map of said geographic information systemand each of said plural pieces of shot images respectively, a parameterfor use in computing a photographic area on the ground surface havingbeen shot is compensated based on a compensation value at the time ofhaving extracted a landmark last, and shot images to be joined partiallyoverlapped with each other and displayed being superposed on the map aremoved such that an overlapped state at said overlap part is of thelargest extent, and thereafter are joined.
 38. The image display methodaccording to claim 31, wherein landmarks are extracted from a map of thegeographic information system and each shot image respectively, aparameter for use in computing each photographic area of the groundsurface having been continuously shot is compensated based on a currentcompensation value, getting back to a halfway point between the shotimage of when a landmark has been extracted last time and the currentshot image, and said plural pieces of shot images are displayed beingsuperposed with high precision on a map of the geographic informationsystem.
 39. The image display method according to claim 35, wherein analtitude compensation value is registered at a point of land wherealtitude compensation processing of a shot image is executed due tocoincidence of the landmarks, and said registered altitude compensationvalue can be utilized again as a reference value of altitudecompensation in the case of flying at a point of land close to saidpoint from the next time on.